Self-referenced tracking

ABSTRACT

A new tracking technique is essentially ”sourceless” in that it can be used anywhere with no set-up, yet it enables a much wider range of virtual environment-style navigation and interaction techniques than does a simple head-orientation tracker. A sourceless head orientation tracker is combined with a head-worn tracking device that tracks a hand-mounted 3D beacon relative to the head. The system encourages use of intuitive interaction techniques which exploit proprioception.

RELATED APPLICATIONS

[0001] This application is a continuation of and claims priority under35 USC §120 to U.S. patent application Ser. No. 09/770,691, filed onJan. 26, 2001, and is entitled under 35 USC § 119(e) to the benefit ofthe filing date of U.S. Provisional Patent Application Serial No.60/178,797, filed on Jan. 28, 2000, the contents of both which arehereby incorporated by reference.

BACKGROUND

[0002] This invention relates to self-referenced tracking.

[0003] Virtual reality (VR) systems require tracking of the orientationand position of a user's head and hands with respect to a worldcoordinate frame in order to control view parameters for head mounteddevices (HMDs) and allow manual interactions with the virtual world. Inlaboratory VR setups, this tracking has been achieved with a variety ofmechanical, acoustic, magnetic, and optical systems. These systemsrequire propagation of a signal between a fixed ”source” and the tracked”sensor” and therefore limit the range of operation. They also require adegree of care in setting up the source or preparing the site thatreduces their utility for field use.

[0004] The emerging fields of wearable computing and augmented reality(AR) require tracking systems to be wearable and capable of operatingessentially immediately in arbitrary environments. ”Sourceless”orientation trackers have been developed based on geomagnetic and/orinertial sensors. They allow enough control to look around the virtualenvironment and fly through it, but they don't enable the”reach-out-and-grab” interactions that make virtual environments sointuitive and which are needed to facilitate computer interaction.

SUMMARY

[0005] In one aspect, in general, the invention provides a new trackingtechnique that is essentially ”sourceless” in that it can be usedanywhere with no set-up of a source, yet it enables a wider range ofvirtual environment-style navigation and interaction techniques thandoes a simple head-orientation tracker, including manual interactionwith virtual objects. The tracker and can be used by novice end userswithout any knowledge of tracking technology, because there is nothingto set up or configure.

[0006] In another aspect, in general, the invention features mounting atracker on a user's head and using the tracker to track a position of alocalized feature associated with a limb of the user relative to theuser's head. The localized feature associated with the limb may includea hand-held object or a hand-mounted object or a point on a hand.

[0007] In another aspect, in general, the invention features mounting asourceless orientation tracker on a user's head and using a positiontracker to track a position of a first localized feature associated witha limb of the user relative to the user's head.

[0008] In another aspect, in general, the invention features tracking apoint on a hand-held object such as a pen or a point on a hand-mountedobject such as a ring or a point on a hand relative to a user's head.

[0009] In another aspect, in general, the invention features using aposition tracker to determine a distance between a first localizedfeature associated with a user's limb and a second localized featureassociated with the user's head.

[0010] In another aspect, in general, the invention features a positiontracker which includes an acoustic position tracker, an electro-opticalsystem that tracks LEDs, optical sensors or reflective marks, a videomachine-vision device, a magnetic tracker with a magnetic source held inthe hand and sensors integrated in the headset or vice versa, or a radiofrequency position locating device.

[0011] In another aspect, in general, the invention features asourceless orientation tracker including an inertial sensor, atilt-sensor, or a magnetic compass sensor.

[0012] In another aspect, in general, the invention features mounting adisplay device on the user's head and displaying a first object at afirst position on the display device.

[0013] In another aspect, in general, the invention features changingthe orientation of a display device, and, after changing the orientationof the display device, redisplaying the first object at a secondposition on the display device based on the change in orientation.

[0014] In another aspect, in general, the invention features determiningthe second position for displaying the first object so as to make theposition of the first object appear to be fixed relative to a firstcoordinate reference frame, which frame does not rotate with the displaydevice during said changing of the orientation of the display device.

[0015] In another aspect, in general, the invention features displayingthe first object in response to a signal from a computer.

[0016] In another aspect, in general, the invention features mounting awearable computer on the user's body, and displaying a first object inresponse to a signal from the wearable computer.

[0017] In another aspect, in general, the invention features displayingat least a portion of a virtual environment, such as a fly-throughvirtual environment, or a virtual treadmill, on the display device.

[0018] In another aspect, in general, the invention features displayinga graphical user interface for a computer on the display device.

[0019] In another aspect, in general, the invention features firstobject being a window, icon or menu in the graphical user interface.

[0020] In another aspect, in general, the invention features the firstobject being a pointer for the graphical user interface.

[0021] In another aspect, in general, the invention features changingthe position of the first localized feature relative to the positiontracker and, after changing the position of the first localized feature,redisplaying the first object at a second position on the display devicedetermined based on the change in the position of the first localizedfeature.

[0022] In another aspect, in general, the invention features displayinga second object on the display device, so that after changing theposition of the first localized feature, the displayed position of thesecond object on the display device does not change in response to thechange in the position of the first localized feature.

[0023] In another aspect, in general, the invention features determiningthe second position so as to make the position of the first objectappear to coincide with the position of the first localized feature asseen or felt by the user.

[0024] In another aspect, in general, the invention features changingthe orientation of the first coordinate reference frame in response to asignal being received by the computer.

[0025] In another aspect, in general, the invention features changingthe orientation of the first coordinate reference frame in response to achange in the position of the first localized feature.

[0026] In another aspect, in general, the invention features changingthe orientation of the first coordinate reference frame in response to asignal representative of the location of the user.

[0027] In another aspect, in general, the invention features changingthe orientation of the first coordinate reference frame in response to asignal representative of a destination.

[0028] In another aspect, in general, the invention features changingthe orientation of the first coordinate reference frame in response to asignal representative of a change in the user's immediate surroundings.

[0029] In another aspect, in general, the invention features changingthe orientation of the first coordinate reference frame is changed inresponse to a signal representative of a change in the physiologicalstate or physical state of the user.

[0030] In another aspect, in general, the invention featuresredisplaying the first object further comprises changing the apparentsize of the first object according to the change in position of thefirst localized feature.

[0031] In another aspect, in general, the invention features mounting aportable beacon, transponder or passive marker at a fixed point in theenvironment and determining the position vector of a second localizedfeature associated with the user's head relative to the fixed point.

[0032] In another aspect, in general, the invention features determiningthe position vector of the first localized feature relative to the fixedpoint.

[0033] In another aspect, in general, the invention features mounting asourceless orientation tracker on a second user's head and determiningthe position of a localized feature associated with the body of thesecond user relative to the fixed point.

[0034] In another aspect, in general, the invention features determiningthe position vector of a second localized feature associated with theuser's head relative to the fixed point without determining the distancebetween the second localized feature and more than one fixed point inthe environment.

[0035] In another aspect, in general, the invention features displayingthe first object at a third position after displaying the first objectat the third position, changing the orientation of the display, andafter changing the orientation of the display, continuing to display thefirst object at the third position.

[0036] In another aspect, in general, the invention features the firstobject being a window in a wraparound computer interface.

[0037] In another aspect, in general, the invention featuresredisplaying the changed position of the first localized feature notbeing within the field of view of the display when the first object isredisplayed.

[0038] In another aspect, in general, the invention features displayingthe first object at a position coinciding with the position of the firstlocalized object when the first localized object is within the field ofview of the display.

[0039] In another aspect, in general, the invention features positioningthe first localized feature at a first point positioning the firstlocalized feature at a second point and calculating the distance betweenthe first point and the second point.

[0040] In another aspect, in general, the invention features determininga position vector of the first localized feature relative to a secondlocalized feature associated with the user's head and modifying theposition vector based on an orientation of the user's head.

[0041] In another aspect, in general, the invention features setting anassumed position for the user's head in a coordinate system and settinga position for the first localized feature in the coordinate systembased on the assumed position of the user's head and said positionvector.

[0042] In another aspect, in general, the invention features measuringthe orientation of the user's head relative to a fixed frame ofreference.

[0043] In another aspect, in general, the invention features setting avirtual travel speed and direction for the user modifying the assumedposition for the user's head based on the user's virtual travel speedand direction.

[0044] In another aspect, in general, the invention features mounting onthe head of a user a three degree of freedom orientation tracker fortracking the orientation of the head, and a three degree of freedomposition tracker for tracking the position of a first localized featureon the user's limb relative to a second localized feature on the user'shead, computing a position vector for the first localized featurerelative to the second localized feature, determining a rotation matrixbased on information received from the rotation tracker, andtransforming the position vector into a position vector for a fixedframe of reference based on the rotation matrix.

[0045] In another aspect, in general, the invention features using anacoustic or radio frequency position tracker to track a position of afirst localized feature associated with a limb of the user relative tothe user's head.

[0046] In another aspect, in general, the invention features mounting avideo camera on the back of the user's head and displaying an imagegenerated by the video camera in a portion of a display device mountedon the user's head.

[0047] In another aspect, in general, the invention features mounting afirst inertial sensor on a user's head, mounting a second inertialsensor elsewhere on the user's body or in an object held by the user,and tracking the position of one inertial sensor relative to the other.

[0048] Some embodiments of the invention include sensing data at thefirst and second inertial sensors and using the sensed data to track theposition of one inertial sensor relative to the other, tracking theposition of the inertial sensor is done without reference to any signalreceived from a source not mounted on or held by the user and correctingthe drift of the relative position or orientation of the second inertialsensor relative to the first inertial sensor by measurements betweendevices on the user's head and devices elsewhere on the users body.

[0049] Among the advantages of the invention are one or more of thefollowing. The device is easy to don, can track both head and hand, addsno new cables to a wearable computer system, works anywhere indoors oroutdoors with no preparation, and is simpler than alternatives such asvision-based self-tracking.

[0050] The details of one or more embodiments of the invention are setforth in the accompanying drawings and the description below. Otherfeatures, objects, and advantages of the invention will be apparent fromthe description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

[0051]FIG. 1 is a perspective view of a self-referenced tracking devicemounted on a head.

[0052]FIG. 2 is a block diagram.

[0053]FIG. 3 is a graph of tracking coverage and relative resolution.

[0054]FIG. 4 is a view of an information cockpit.

[0055]FIG. 5 shows a user using a virtual reality game.

[0056] Like reference symbols in the various drawings indicate likeelements.

DETAILED DESCRIPTION

[0057] As seen in FIG. 1, implementations of the invention may combine asourceless head orientation tracker 30 with a head-worn tracking device12 that tracks a hand-mounted 3D beacon 14 relative to the head 16. Oneimplementation uses a wireless ultrasonic tracker 12, which has thepotential for low cost, lightweight, low power, good resolution, andhigh update rates when tracking at the relatively close ranges typicalof head-hand displacements.

[0058] As FIG. 1 illustrates, this arrangement provides a simple andeasy to don hardware system. In a fully integrated wearable VR systemusing this tracker there are only three parts (a wearable computer 10, aheadset 15 with an integrated tracking system, and a hand-mounted beacon14) and one cable connection 18. This is possible because the entireultrasonic receiver system 12 for tracking the beacon can be reduced toa few small signal-conditioning circuits and integrated with thesourceless orientation tracker 30 in the head-worn display 15. Bysharing the microprocessor and its power and communications link to thewearable, the cost and complexity are reduced.

[0059] The benefits of this combination of elements stem from theserealizations:

[0060] 1. It is usually not important to track the hand unless it is infront of the head. Thus range and line-of-sight limitations are noproblem if the tracker is mounted on the forehead.

[0061] 2. The hand position measured in head space can be transformedinto world space with good seen/felt position match using an assumedhead pose, no matter how inaccurate.

[0062] 3. Using one fixed beacon, the same tracking hardware can providefull 6-DOF tracking.

[0063] Implementations of the invention may exhibit:

[0064] 1. A new tracking concept that enables immersive visualizationand intuitive manual interaction using a wearable system in arbitraryunprepared environments.

[0065] 2. An information cockpit metaphor for a wearable computer userinterface and a set of interaction techniques based on this metaphor.

[0066] As shown in FIG. 2, a simple proof-of-concept implementationcombines an InterSense IS-300 sourceless inertial orientation tracker 40(available from InterSense, Inc., in Burlington, Mass.) with a PegasusFreeD ultrasonic position tracker 50 (available from PegasusTechnologies Ltd. in Holon, Israel). The IS-300 has an ”InertiaCube”inertial sensor assembly 42, just over an inch on a side, cabled to asmall computational unit 44 that outputs orientation data through aserial port 46. The FreeD product consists of a finger-worn wirelessultrasonic emitter 50A with two mouse buttons 54, and an L-shapedreceiver bar 50B which normally mounts on the frame of a computermonitor, and outputs x,y,z data through a serial port. For ourexperiments we mounted the InertiaCube and the L-shaped receiver bar onthe visor 60 of a V-Cap 1000 see-through HMD (available from VirtualVision of Seattle, Wash.). The FreeD therefore measures the ringposition relative to the head-fixed coordinate frame whose orientationwas measured by the IS-300.

[0067] Data from both trackers is transmitted to a PC 62 (Pentium 300MHz, Windows 98) running a program 63 that uses Windows DirectX andDirect3D capabilities to display graphics and effect interactiontechniques. The graphics output window of Direct3D is maximized to takecontrol over the entire screen, and VGA output 64 (640×480 at 60 Hz) ispassed into the V-Cap HMD as well as a desktop monitor.

[0068] The program 63 includes a tracker driver 71 and a fairlyconventional VR rendering environment 72 that expects to receive 6-DOFhead and hand tracking data from the tracker driver as well as buttonstates 65 for the hand tracking device. The interaction techniques to bedescribed are implemented in the tracker driver. The basic functions ofthe tracker driver, when tracking a single 3-DOF point on the hand, are:

[0069] 1. Read in and parse the orientation data 68 from the IS-300 andthe position triad 70 from the FreeD.

[0070] 2. Package the orientation data with the current head position inworld-frame, and output the combined 6-DOF data record 73 for the headto the VR program. The current assumed world-frame head position is thesame as the previous one unless the user is in the process of performinga navigation interaction such as flying. In this case the position isincremented based on the flying speed and direction.

[0071] 3. Transform the hand position vector from head frame to worldframe by first multiplying by the rotation matrix from head to worldframe obtained from the orientation tracker, then adding the currentassumed world-frame head position. Output the result to the VR programas a 3-DOF position record 74 for the hand device.

[0072] The simple implementation just described is wearable, but cannotbe integrated into an HMD elegantly, largely due to the size and powerconsumption of the IS-300 processing unit. A low-cost wearable versionusing available technologies could be implemented as follows:

[0073] The core of this implementation is an inertial head orientationmodule called InterTrax 2 (available from InterSense and designed foruse with consumer HMDs such as the Sony Glasstron and Olympus EyeTrek).Using tiny piezoelectric camcorder gyros, and solid-state accelerometersand magnetometers, InterTrax 2 is designed as a single long narrowcircuit board 30 (FIG. 1) to lie across the top of the head mounteddisplay unit along the brow line. It is 9 cm long, 2 cm wide, and 0.5 cmthick with all components, except for a vertical gyro in the center,which sticks up 1 cm higher. It contains a low-power embedded 16-bitprocessor that runs a simplified fixed-point version of the GEOSdrift-corrected orientation-tracking algorithm used in the IS-300. Itcommunicates to the host through a single USB connector through which itdraws its power, and can be manufactured for very low cost in volume. Itis expected to achieve accuracy on the order of 2-3°, which issufficient because the accuracy with which the hand avatar follows thephysical hand is totally independent of orientation tracking accuracy.

[0074] Another component is an embedded ultrasonic rangefinder (perhapsbased on the Pegasus FreeD technology). As shown in FIG. 1, threemicrophones 80, 82, 84 and their ultrasonic pulse detection circuitstogether with the InterTrax 2 board are embedded in a rigid plasticassembly designed to fit elegantly over the brow of an HMD. (In someembodiments, all components would be embedded inside the HMD displayunit while sharing the HMD's cable 18, but in others, the addedcomponents are clipped on) The InterTrax 2 processor has enough unusedtimer inputs and processing bandwidth to timestamp the signals from thethree ultrasonic pulse detectors and relay this data down its USB link.

[0075] The ultrasonic tracking technology can be modified to takeadvantage of the very short range requirements. First, ultrasonicfrequency may be increased from 40 KHz to a higher frequency. Thisincreases the attenuation in air, and virtually eliminates reverberationand interference between nearby users. Second, the system can takeadvantage of the much reduced reverberation and the short time-of-flightto increase the update rate of tracking to, say, 240 Hz, thus allowingthe system to average 4 position samples for each 60 Hz graphics update,or track up to 4 beacons at 60 Hz. To calculate the resolution that thiswould yield in various parts of the tracking volume we calculated theGeometric Dilution of Precision (GDOP) throughout the tracking volumegiven the intended geometry of the microphone mounts on the headset. Theintended headset geometry, tracking range and optical field of view areillustrated superimposed on an isogram of a vertical slice through theGDOP data in FIG. 3. The plane of the microphones is angled downward 45°to insure that the system has tracking coverage for hands in the lap.The resolution at any point in space is the range measurement resolution(about 0.1 mm for short range ultrasonic measurements using 40 KHz)multiplied by the GDOP value, divided by 2 as a result of the 4×oversampling and averaging. Thus the expected resolution isapproximately 0.5 mm at a distance of 400 mm away from the headset.

[0076] A goal of a wearable computer is to keep the user's hands free toperform tasks. For this reason, the system uses a wireless 3-DOF ringpointer for interaction. The FreeD ring-mouse previously described isapproximately the right size. In some implementations of the system, thetracker will need to be triggered by a unique IR code from the headset,so that multiple beacons can be tracked.

[0077] In interactive visualization and design (IVD) and many other VRapplications, a pen-style input device may be more useful. Animplementation could use a wireless 5-DOF pen using the same basictechnology as the 3-DOF ring pointer, but employing two emitters thatare activated in an alternating sequence. A compact omni-directional pencould be implemented using cylindrical radiating ultrasonic transducersthat have been developed by Virtual Ink (Boston, Mass.), mounted at theends of a cylindrical electronics unit approximately the size of anormal pen, with two mouse buttons.

[0078] An additional device that could be included in the system andwhose applications are discussed below is a small wireless anchor beaconthat can be easily stuck to any surface. Ultrasonic beacons fromInterSense are of suitable size and functionality.

[0079] Portable VR Application

[0080] Object Selection and Manipulation Exploiting Proprioception

[0081] M. Mine, F. Brooks, and C. Sequin. (Moving Objects in Space:Exploiting Proprioception in Virtual Environment Interaction. InSIGGRAPH 97 Conference Proceedings, ACM Annual Conference Series,August, 1997), have discussed the benefits of designing virtualenvironment interaction techniques that exploit our proprioceptive senseof the relative pose of our head, hands and body. A variety oftechniques were presented, such as direct manipulation of objects withinarms reach, scaled-world grab, hiding tools and menus on the users body,and body-relative gestures.

[0082] Implementations of the invention have advantages overconventional world-frame tracking systems for implementing thesetechniques effectively. With conventional trackers, any error in headorientation tracking will cause significant mismatch between the visualrepresentation of the virtual hand and the felt position of the realhand, making it difficult to accurately activate hidden menus while thevirtual hand is not in view. With implementations of the invention, thehead orientation accuracy is immaterial and visual-proprioceptive matchwill be good to the accuracy of the ultrasonic tracker—typically 1-2 mm.

[0083] Locomotion & View Control Tricks

[0084] This section describes a few techniques to permit user locomotionand view control.

[0085] Flying and Scaled-World Grab

[0086] The usual navigation interface device in fly-through virtualenvironments is a joystick. This is appropriate for a flight simulator,but reduces one's sense of presence in terrestrial environments, whereturning one's body toward the destination is more instinctive thanturning the world until the destination is in front. Implementations ofthe invention support this more immersive type of flying. No matter howone turns, if she raises a hand in front of her it will be trackable,and can be used to control flight speed and direction. Better yet, shecan use two-handed flying, which can be performed with the arms in arelaxed position and allows backwards motion, or the scaled-world grabmethod to reach out to a distant object and pull oneself to it in onemotion.

[0087] Walking Using Head Accelerometers as a Pedometer

[0088] For exploratory walk-throughs, the sense of presence is greatestfor walking, somewhat reduced for walking-in-place, and much furtherreduced for flying. M. Slater, A. Steed and M. Usoh (The VirtualTreadmill: A Naturalistic Metaphor for Navigation in Immersive VirtualEnvironments. In First Eurographics Workshop on Virtual Reality, M.Goebel Ed. 1993), and M. Slater, M. Usoh and A. Steed (Steps and Laddersin Virtual Reality. In Proc. Virtual Reality Software & Technology 94,G. Singh, S. K. Feiner, and D. Thalmann, Eds. Singapore: WorldScientific, pages 45-54, August 1994) have described a ”virtualtreadmill” technique in which a neural network is trained to recognizethe bouncing pattern of a position tracker on an HMD, and thus controlvirtual motion. Inertial head-orientation trackers do not normallyoutput the position obtained by double integrating the accelerometers,because it drifts too much to be useful, but it seems reasonable thatpattern analysis of the acceleration signals would produce good results.

[0089] Head-Motion Parallax Using Anchor Beacon

[0090] When working with close objects, head motion parallax is animportant visual cue. It can be achieved with the tracking system of theinvention on demand by using a trick. Normally, the system uses the3-DOF position vector from the user's head to the hand-mounted beacon totrack the position of the hand relative to the head, maintaining thehead location fixed. When desired, the user may hold the hand still (sayon a desk), and push a button to reverse this process, so that thetracker driver interprets the negative of the measured vector (in worldframe) as a position update of the head relative to the stationary hand.He can then move his head back and forth to look around an object, andrelease the button when his viewpoint is repositioned for optimalviewing. After flying or walking to an area, this may be a convenientway of making finely controlled viewpoint adjustments using natural neckmotion. Note that this operation is equivalent to grabbing the world andmoving it around with one's hand, which may be a more convenientmaneuver while standing.

[0091] Implementations of the invention can perform full 6-DOF headtracking using only one fixed reference point in the environment, whilemost acoustic and optical trackers require at least three. This works inthe invention because head orientation is completely constrained by thesourceless head-tracker. This observation suggests another interestingtrick. One may carry an extra wireless anchor beacon in a pocket andplace it down on the table or stick it to a wall near a work area.Within range of this beacon, he can enjoy full 6-DOF tracking of bothhead and hand.

[0092] Wearable Computing Information Cockpit Interface

[0093] Information Cockpit Metaphor

[0094] In the field of wearable computing, three modes of displayingobjects in a head-mounted display have been discussed. Head-stabilizedobjects are displayed at a fixed location on the HMD screen, so theymove with your head motion and require no tracking. World-stabilizedobjects are fixed to locations in the physical environment. To causethem to stay fixed despite user head-motion requires full 6-DOF headtracking. Body-stabilized objects are displayed at a fixed location onthe information surround, a kind of cylindrical or spherical bubble ofinformation that follows the user's body position around. Headorientation tracking allows the user to look at different parts of thesurroundings by turning his head, but position tracking is not needed.

[0095] Pure head-stabilized displays are usually used with small opaquemonocular monitors mounted off to the side of the user's field of view.Without head tracking, this is better than having a display directly infront of the eye with information constantly blocking the frontal view.Use of this paradigm is widespread, and most of the wearable computervendors provide this style of untracked sidecar display. This is roughlyequivalent to wearing your desktop computer on your belt with themonitor mounted on a headband so that it is always available forhands-free viewing.

[0096] At the other end of the spectrum are world-stabilized ARdisplays, which must be implemented using see-through optics placeddirectly in front of the eyes. For a variety of applications such assurgery, construction and maintenance, this is a highly valuablecapability. However, it requires sophisticated tracking and calibration,and is likely to remain a high-end subset of the total wearablecomputing market for quite a few years.

[0097] In the middle ground of complexity are the less commonbody-stabilized displays, which also tend to be implemented with seethrough HMDs. As implemented by S. Feiner, B. MacIntyre, M. Haupt, andE. Solomon (Windows on the World: 2D Windows for 3D Augmented Reality.In Proc. ACM UIST 93. ACM Press, November 1993) objects were drawn on a170° horizontal by 90° vertical portion of a sphere. To prevent userdisorientation, this hemispherical ”virtual desk” was kept in front ofthe user's body by mounting an additional orientation tracker on theuser's torso, and using the difference between the head yaw and torsoyaw to pan the viewport. The desk was thus slaved to the user's torso,and the user could easily locate windows on it using his innateknowledge of head turn relative to the torso. This is intuitive but hasthe drawback that an additional orientation sensor must be mounted onthe user's torso. This adds cost, makes the system more difficult todon, and causes the virtual desk to shift around in response to slightpostural shifting of the user's torso, wobbling of the sensor mount, ormetallic distortion of the relative magnetic heading between the twosensors. An implementation of the invention uses a variation on thistheme, based on an ”information cockpit” metaphor instead of abody-stabilized desk.

[0098] The information cockpit consists of a clear windshield,optionally drawn as a thin wireframe border, and a cluster of virtualinstruments around it. As with the body-stabilized technique, the user'shead is always in the center of the cockpit, but the heading directionof the cockpit stays fixed until the user changes it. Generally, theuser first positions the windshield towards the objects he will beworking on with his hands, and keeps the windshield area fairly clear ofaugmentations so that he can see what he is doing. Thereafter, the usercan turn to look at the instruments, with or without turning his torso,and the instruments will not move. To prevent the user from becomingdisoriented or being forced to strain his neck as he moves around, theimplementation provides the user with steering techniques.

[0099] Outdoor Navigation Application

[0100]FIG. 4 shows an example of an information cockpit for an outdoornavigation application. The active field-of-view of the see-through HMDis indicated by heavy black rectangle 400. Thus only the augmentationswithin this rectangle are visible to the user, but rotating the headmoves this active view port around the scene and reveals the otheraugmentations once they are inside of it. In this example there are afew frequently-used icons 401 that are fixed (i.e. head-stabilized) inthe upper right of the heads-up display that will always be visible.There are additional icons 402 in the dashboard that are stabilized tothe information cockpit, and therefore can only be seen when the userlooks down a little to check them. Some of these are miniatureinformation instruments, such as dials and gauges, while others areicons used to bring up larger information instruments such as a webbrowser or interactive map display. By clicking on the map icon on thedashboard, the full-size map application window 404 pops up in themiddle of the active display area. The user may either quickly examineit then minimize it again, or save it for on-going reference by fixingit to a convenient spot on the information cockpit ”windshield” 410 ashas been done in FIG. 4. The user can see a corner of the map in thecurrent view, but can look at the whole map again by looking up and tothe right. Virtual rear view mirrors 406 (fed by a video camera on theback of the head) have likewise been placed in three locations on thevirtual cockpit, but the user can re-position or close any of these fourinformation instruments at any time. In this example, the headingdirection of the cockpit is controlled by the application in order toguide the user to a destination. Using a GPS receiver in the user'swearable computer, the application orients the cockpit along thedirection from the user's current position to the destination, so heneed only follow the dotted lines 408 to their vanishing point on thehorizon to walk in the correct direction. This provides a virtualsidewalk in the forest, much as pilots are guided by virtualtunnel-in-the-sky displays. In an urban setting, the computer would usemap correlation to orient the cockpit along the current road in thesuggested walking direction.

[0101] Steering and Interaction

[0102] The ring tracker can be used for several purposes in wearablecomputer applications: direct pointing to objects, virtual mouse padcursor control, command gestures, and measuring or digitizing.

[0103] Direct pointing to objects

[0104] When the ring tracker enters the viewing frustum of the HMD, thecursor jumps to the location of the ring and follows it. This providesrapid direct selection of objects, taking full advantage of naturaleye-hand coordination. In the virtual cockpit, one may glance up fromthe windshield to a side panel, see an instrument he wants to use, reachout to exactly where he sees it and click on it with one of the ringbuttons to activate it or drag it into another view.

[0105] Many useful operations can be accomplished most easily withdirect selection and manipulation of objects. You can move and resizewindows (i.e. instruments) the usual 2D way by dragging their borders.However, you can also exploit the 3D tracking of the ring tosimultaneously move and resize an instrument. Simply grab the title barand pull it toward you to make it larger or away from you to make itsmaller, while simultaneously positioning it. If you pull it in towardsyour head far enough, as if to attach it to your HMD, it will changecolors, indicating that if you let go of it, it will remain as ahead-stabilized object. This is effectively like grabbing an instrumentoff your cockpit panel and attaching it to your Heads-Up-Display (HUD)so that it will always be visible in the foreground no matter where youlook. By pushing it away far enough it will convert back to a cockpitpanel instrument.

[0106] One of the cockpit windows that can be manipulated in a similarmanner is the windshield itself. Simply click on any clear area of the”glass” where there aren't any graphical objects you might accidentallyselect, then drag it left/right or up/down to rotate the whole cockpitin space. This is one way of”steering” the cockpit, which isparticularly useful for small course corrections or size adjustments orto refocus your attention on another area of the workbench nearby.

[0107] Virtual mouse pad cursor control

[0108] Though fast and intuitive, the direct pointing technique wouldbecome very tiring if used to work with an instrument that requiresextended repetitive clicking, such as a web browser or hypertext manual.A virtual mouse pad technique can overcome this problem. As soon as theuser's hand drops below the viewing frustum of the HMD, the cursorcontrol automatically switches into this mode, in which left-and-rightmotion of the ring moves the cursor left-and-right, in-and-out motionmoves it up and down, and vertical position has no effect. This allowsthe user to rest his hand comfortably in his lap or on a desk, andcontrol the cursor by sliding his hand horizontally a few inches as ifon an imaginary mouse pad.

[0109] It is desirable that if the user positions the cursor on aparticular object then moves his head without moving the ring, thecursor will remain on the object. This means that the cursor is drawn asan object in the cockpit-stabilized coordinates rather than thehead-stabilized screen coordinates. This has several implications.First, the cursor is associated with a point on the sphericalinformation cockpit surface, only a portion of which is visible in theHMD, so the cursor could be out of view and quite difficult to find. Awiggling gesture is then used to bring it back into the current centerof display. Second, the ring tracking must be calculated in the cockpitstabilized coordinate frame, which means that if the user turns to theright, an ”in-and-out” motion switches from cockpit x-axis to y-axis andhas an unexpected effect. To avoid this, the ring position istransformed into cylindrical polar coordinates and the radial andtangential components are used to control cursor vertical and horizontalmotion respectively.

[0110] Command Gestures

[0111] Ring tracker gestures may be used as a substitute for voicecommands in situations where visual theatrics are more acceptable thanaudible ones, or where it is too noisy for reliable speech recognition.In general, gestures should commence outside of the direct pointing andvirtual mouse pad regions, in order to avoid accidentally selecting andmoving objects. This leaves the sides and top of the viewing frustum,and the first few inches in front of the face (which are not used fordirect pointing). The gestures are executed by depressing a mousebutton, possibly making a certain movement, then releasing the button.They are always relative to the head in order to exploit proprioception,and the fact that the head is tracked, while the rest of the body isnot. Many gestures may be defined, but the most commonly needed is aboresight command to reset the heading direction of the cockpit to thecurrent forward direction of the person's head as he walks about.

[0112] Measuring or Digitizing

[0113] Most people can hold their head very still, which opens thepossibility that the ring tracker can be used to make measurementsbetween two points that are close enough that both can be seen withoutmoving the head. This might be useful in an application such as takinginventory of how many pieces of each size are in a stockroom. Likewise,ail application might ask you to quickly digitize a few corners of acomponent so it can determine based on the dimensions what model of thecomponent you are looking at and locate the appropriate manual pages.

[0114] To measure the distance between two close objects that are bothwithin the display FOV at the same time, the user clicks both objectswhile holding his head still. The distance is computed as the norm ofthe difference of the two vector positions thus stored.

[0115] For two objects that are too far apart to be in the display FOVat once, a more elaborate procedure may be employed. The user firstlooks at the first object, positions the pointer beacon on it anddepresses a button. At the moment the button is pressed, a world frameposition vector (p1) of the first object is stored and then the trackingmode is switched to 6-DOF tracking of the head relative to thestationary hand-held pointer, as previously described. While holding thepointer stationary on the object and keeping the button depressed, theuser then repositions his head until the second object is in view,releases the button, and holds his head still while moving the pointerto the second object, then clicking it to capture the second positionvector (p2) in the same world coordinate frame as the first. Thistechnique may be practiced either with a single pointing beacon operatedby one hand, or using separate pointing beacons in each hand, to achieveapproximately the same functionality as a conventional tape measure, butwith the added benefit that the measurements are automatically stored ona digital computer.

[0116] Relationships among remote objects may also be measured usingstandard triangulation surveying methods, exploiting the functionalsimilarity of a see-through HMD optic with orientation tracker to asurveyor's theodolite (although a tripod mounted theodolite is likely tobe more accurate).

[0117] Mixed Display and AR-on-Demand Applications

[0118] The previous section presented the information cockpit as aspecific variation on Feiner's body-stabilized information surround.However, the cockpit metaphor also allows the user to make use of thehead-stabilized and world-stabilized coordinate frames at the same time.The previous section gave one example of this in which the pilot dragsinformation from the cockpit onto the HUD, which makes ithead-stabilized. For example, one may wish to have an alerting devicealways visible in the HUD that pops up notifications whenever a phonecall, page or email is received, or when a scheduled meeting is about tobegin, etc.

[0119] Likewise, one may wish to grab a certain instrument and paste itonto a physical object in worldspace. For example, while debugging acircuit board, you could overlay an interactive block diagram orschematic on the board, and attach a virtual scope trace to your handthat is holding the scope probe (which is possible because the hand istracked by the ring pointer). To do this, you must first plant an anchorbeacon, then click three corners of the circuit board to align the blockdiagram to it.

[0120] One important reason to plant anchor beacons is to create ashared AR workspace for communication or collaboration with coworkers asdescribed in M. Billinghurst, S. Weghorst and T. Furness (Shared Space:An Augmented Reality Approach for Computer Supported Cooperative Work.Virtual Reality Vol. 3(1) 1998) and D. Schmalstieg, A. Fuhrmann, Z.Szalavari, and M. Gervautz (Studierstube: An Environment forCollaboration in Augmented Reality. In CVE 96 Workshop Proceedings,September, 1996) incorporated by reference. Imagine a paperlessconstruction site with numerous workers building a structure accordingto the plans they are viewing on their wearable computers. It is nicethat they don't have to drag large rolls of blueprints around, but theyhave no way to stand around a blueprint and point to things. Thesolution is for someone to drop two anchor pins on a table, defining thetop two corners of a virtual blueprint or model that each person can seein correct perspective from his own vantage point.

[0121] A Variant Technique for Tracking the User's Hand

[0122] Some implementations of the invention use an inertial orientationsensor to track the rotation of the head, and an acoustic or opticalposition tracker to track the position of the hand relative to the head.For many applications, the performance of the acoustic or opticalposition tracker is sufficient. Furthermore, it has the great advantagethat the item being tracked can be a small wireless transponder, or evena passive marker. For some applications, such as the ring-mountedpointing device for wearable computing, this is an overwhelmingadvantage.

[0123] However, for some applications, such as a virtual reality game,it may be desired to have the virtual object controlled by the handtracker (e.g. a virtual sword or gun or racquet) respond to the handmotion with extremely fast smooth response. Acoustic, magnetic, orvideometric hand trackers may introduce noticeable latency or jitter inthese applications. Inertial position and orientation trackers are wellknown to provide extremely low latency and low jitter, but they requiredrift correction, especially if tracking position and not justorientation is desired. In a typical virtual reality application, theuser's head and hand may both be tracked with 6 degrees of freedomrelative to an external reference frame by using inertial sensors on thehead and on the hand to measure their motion with a high update rate andlow latency. The drift of these inertial sensors is corrected by makingmeasurements with an ultrasonic, optical or magnetic tracking referencedevice mounted in the environment.

[0124] In some implementations of the present invention, the drift andlatency issues can be addressed without the requirement of a referencedevice mounted in the environment. Foxlin, ”Head-tracking Relative to aMoving Vehicle or Simulator Platform Using Differential InertialSensors,” Proceedings of Helmet and Head-Mounted Displays V, SPIE vol.4021 (2000) and co-pending U.S. patent application Ser. No. 09/556,135,which are incorporated herein by reference, describe techniques whichenable the use of inertial sensors to track the motion of an objectrelative to a reference frame that is moving, even where the motion isnot known completely. These techniques require that inertial sensors beattached to the moving body being used as the reference frame (in thecited references an example of the reference frame is given of a vehicleor motion-platform and an example of the tracked object is given as ahead; in the present invention, the moving reference frame may be theuser's head and the tracked object may be the user's hand orhand-mounted or hand-held object), as well as to the object beingtracked (here, e.g., the user's hand). The techniques utilize angularrate and linear acceleration signals from the sourceless orientationtrackers on the reference frame and on the tracked object to derive adifferential inertial signal representative of the motion of the objectrelative to the frame. In embodiments of the present invention, thistechnique may be used to derive a differential inertial signalrepresentative of the motion of the hand relative to the head.

[0125]FIG. 5 illustrates a user wearing a portable VR tennis game ortraining system. The computer and batteries are contained in backpack502, which is cabled to HMD 500 to which are mounted inertial sensors506 and ultrasonic transducers 510. He is holding a hand-held object516, in this case a tennis racquet, to which are attached inertialsensors 508 and ultrasonic transducers 512. These hand-mounted devicesmay be powered by their own batteries and communicate by wireless meansto the system on the users head and torso, or there may be an additionalcable between the racquet and the backpack. The signals from inertialsensors 506 are processed by a first algorithm, preferably adrift-corrected inertial orientation tracking algorithm such asdescribed in U.S. Pat. No. 5,645,077 to obtain a sourceless measurementof the head orientation. In addition, the signals from the hand-mountedinertial sensors 508 and the head-mounted inertial sensors 506 arejointly processed to track both the position and orientation of the handrelative to the head, preferably using an algorithm such as described inFoxlin (2000) and co-pending U.S. patent application Ser. No.09/556,135. The drift of this relative inertial tracking is corrected bythe relative range measurements 514. In the illustrated system there arealso earphones 504 to provide 3D spatialized audio, and a hapticfeedback device 518 to provide tactile feedback to the user when thevirtual ball has hit the virtual racquet.

[0126] In general such a system may be used for other types ofactivities, such as a sword-fighting or gun-fighting game or trainer, asurgical trainer, an immersive design environment, a human-computerinterface, or any other application known or not yet known whichrequires tracking of a user's head and one or more limbs or limb-mounteddevices. While it is especially advantageous for mobile or portableapplications in which the computer is wearable, this is not arequirement, and the user may be cabled to an off-body computer orcommunicate with an off-body computer through a wireless connection. Inthis case, it is still an advantage of the current invention that thetracking is accomplished without setting up an off-body referencedevice.

[0127] Other embodiments are within the scope of the claims.

[0128] The implementations described above track the hand with ahead-mounted acoustic tracking system because this technology can betotally embedded in a lightweight headset and achieve high resolutiontracking over a very wide FOV.

[0129] However, the head mounted position tracker need not be acoustic.It may be an electro-optical system which tracks LEDs, optical sensors,or reflective markers, or a video machine-vision device that recognizesthe hands or fingers or some special markers mounted on the hands orfingers or handheld object, or even a magnetic tracker with a magneticsource held in the hand and sensors integrated in the headset or viceversa, or an RF position locating device.

[0130] The implementation described above use inertial sourcelessorientation trackers. Other implementations may use other forms of headorientation trackers, including trackers based on tilt-sensing ormagnetic compass sensors, or any other form of head orientation tracker.In fact, some implementations may use no head orientation tracker. Inthis case, the tracking system would not enable the user to look aroundin a virtual environment by turning his head, but it would still beuseful for manual interaction with computers using head-worn displays.

[0131] A number of embodiments of the invention have been described.Nevertheless, it will be understood that various modifications may bemade without departing from the spirit and scope of the invention.

What is claimed is:
 1. A method comprising mounting a sourcelessorientation tracker on a user's head, and using a position tracker totrack a position of a first localized feature associated with a limb ofthe user relative to the user's head.
 2. The method of claim 1 in whichthe first localized feature associated with the limb comprises a pointon a hand-held object or a point on a hand-mounted object or a point ona hand.
 3. The method of claim 2, wherein the first localized feature ison a stylus-shaped device.
 4. The method of claim 2, wherein the firstlocalized feature is on a ring.
 5. The method of claim 1 furthercomprising using the position tracker to determine a distance betweenthe first localized feature and a second localized feature associatedwith the user's head.
 6. The method of claim 1 in which the positiontracker comprises an acoustic position tracker.
 7. The method of claim 1in which the position tracker comprises an electro-optical system thattracks LEDs, optical sensors or reflective marks.
 8. The method of claim1 in which the position tracker comprises a video machine-vision devicethat recognizes the first localized feature.
 9. The method of claim 1 inwhich the position tracker comprises a magnetic tracker with a magneticsource held in the hand and sensors integrated in the headset or viceversa.
 10. The method of claim 1 in which the position tracker comprisesa radio frequency position locating device.
 11. The method of claim 1 inwhich the sourceless orientation tracker comprises an inertial sensor.12. The method of claim 1 in which the sourceless orientation trackercomprises a tilt-sensor.
 13. The method of claim 1 in which thesourceless orientation tracker comprises a magnetic compass sensor. 14.The method of claim 1 further comprising: mounting a display device onthe user's head; and displaying a first object at a first position onthe display device.
 15. The method of claim 14 further comprising:changing the orientation of the display device; and after changing theorientation of the display device, redisplaying the first object at asecond position on the display device based on the change inorientation.
 16. The method of claim 15, wherein the second position isdetermined so as to make the position of the first object appear to befixed relative to a first coordinate reference frame, which frame doesnot rotate with the display device during said changing of theorientation of the display device.
 17. The method of claim 16, whereinthe first object is displayed in response to a signal from a computer.18. The method of claim 17, further comprising: mounting a wearablecomputer on the user's body, and wherein the first object is displayedin response to a signal from the wearable computer.
 19. The method ofclaim 15, further comprising displaying a portion of a virtualenvironment on the display device.
 20. The method of claim 19, furthercomprising: displaying a portion of the virtual environment on thedisplay device before changing the orientation of the display device,and displaying a different portion of the virtual environment on thedisplay device after changing the orientation of the display device. 21.The method of claim 19, in which the virtual environment is afly-through virtual environment.
 22. The method of claim 19, in whichthe virtual environment includes a virtual treadmill.
 23. The method ofclaim 15, further comprising displaying a graphical user interface for acomputer on the display device.
 24. The method of claim 23, wherein thefirst object is a window, icon or menu in the graphical user interface.25. The method of claim 23, wherein the first object is a pointer forthe graphical user interface.
 26. The method of claim 16, furthercomprising: changing the position of the first localized featurerelative to the position tracker; and after changing the position of thefirst localized feature, redisplaying the first object at a secondposition on the display device determined based on the change in theposition of the first localized feature.
 27. The method of claim 26,further comprising: displaying a second object on the display device,wherein after changing the position of the first localized feature, thedisplayed position of the second object on the display device does notchange in response to the change in the position of the first localizedfeature.
 28. The method of claim 26, wherein the second position isdetermined so as to make the position of the first object appear tocoincide with the position of the first localized feature as seen orfelt by the user.
 29. The method of claim 17, further comprising:changing the orientation of the first coordinate reference frame inresponse to a signal being received by the computer.
 30. The method ofclaim 29, wherein the orientation of the first coordinate referenceframe is changed in response to a change in the position of the firstlocalized feature.
 31. The method of claim 29, wherein the orientationof the first coordinate reference frame is changed in response to asignal representative of the location of the user.
 32. The method ofclaim 29, wherein the orientation of the first coordinate referenceframe is changed in response to a signal representative of adestination.
 33. The method of claim 29, wherein the orientation of thefirst coordinate reference frame is changed in response to a signalrepresentative of a change in the user's immediate surroundings.
 34. Themethod of claim 29, wherein the orientation of the first coordinatereference frame is changed in response to a signal representative of achange in the physiological state or physical state of the user.
 35. Themethod of claim 27, wherein redisplaying the first object furthercomprises changing the apparent size of the first object according tothe change in position of the first localized feature.
 36. The method ofclaim 1, further comprising: mounting a portable beacon, transponder orpassive marker at a fixed point in the environment; and determining theposition vector of a second localized feature associated with the user'shead relative to the fixed point.
 37. The method of claim 36, furthercomprising determining the position vector of the first localizedfeature relative to the fixed point.
 38. The method of claim 36, whereinthe position vector is determined without determining the distancebetween the second localized feature and more than one fixed point inthe environment.
 39. The method of claim 36, wherein the position vectoris determined without determining the distance between the secondlocalized feature and more than two fixed points in the environment. 40.The method of claim 36, further comprising: mounting a sourcelessorientation tracker on a second user's head; and determining theposition of a localized feature associated with the body of the seconduser relative to the fixed point.
 41. The method of claim 16, furthercomprising: displaying the first object at a third position; afterdisplaying the first object at the third position, changing theorientation of the display; and after changing the orientation of thedisplay, continuing to display the first object at the third position.42. The method of claim 27, wherein the first object is a window in awraparound computer interface.
 43. The method of claim 26, wherein saidchanged position of the first localized feature is not within the fieldof view of the display when the first object is redisplayed.
 44. Themethod of claim 43, further comprising: displaying the first object atan apparent position coinciding with the position of the first localizedobject when the first localized object is within the field of view ofthe display.
 45. The method of claim 1, further comprising: positioningthe first localized feature at a first point; positioning the firstlocalized feature at a second point; and calculating the distancebetween the first point and the second point.
 46. The method of claim 1,further comprising: determining a position vector of the first localizedfeature relative to a second localized feature associated with theuser's head; and transforming the position vector based on anorientation of the user's head.
 47. The method of claim 46, furthercomprising: setting an assumed position for the user's head in acoordinate system; and setting a position for the first localizedfeature in the coordinate system based on the assumed position of theuser's head and said position vector.
 48. The method of claim 47, wheresetting a position for the first localized feature further comprises:measuring the orientation of the user's head relative to a fixed frameof reference.
 49. The method of claim 47, further comprising: setting avirtual travel speed and direction for the user; and modifying theassumed position for the user's head based on the user's virtual travelspeed and direction.
 50. The method of claim 1, wherein the sourcelessorientation tracker comprises a first inertial sensor, and furthercomprising: mounting a second inertial sensor elsewhere on the user'sbody or in an object held by the user; and tracking the position of oneinertial sensor relative to the other.
 51. The method of claim 14,further comprising: mounting a video camera on the back of the user'shead; and displaying an image generated by the video camera in a portionof the display device.
 52. A method comprising: using acoustic or radiofrequency signals to track a position of a first localized featureassociated with a limb of the user relative to the user's head.
 53. Atracking system comprising: an acoustic or radio frequency positiontracker adapted for mounting on a user's head, said tracker beingadapted to track a position of a first localized feature associated witha limb of the user relative to the user's head.
 54. A tracking systemcomprising a sourceless orientation tracker for mounting on a user'shead, and a position tracker adapted to track a position of a firstlocalized feature associated with a limb of the user relative to theuser's head.
 55. A method comprising: mounting a motion tracker on auser's head; using a position tracker to track a position of a firstlocalized feature associated with a limb of the user relative to theuser's head; positioning the first localized feature at a first point;positioning the first localized feature at a second point; andcalculating the distance between the first point and the second point.56. A system comprising: mounting a first inertial sensor on a user'shead; mounting a second inertial sensor elsewhere on the user's body orin an object held by the user; and tracking the position of one inertialsensor relative to the other.
 57. The method of claim 56, furthercomprising: sensing data at the first and second inertial sensors andusing the sensed data to track the position of one inertial sensorrelative to the other.
 58. The method of claim 57, wherein tracking theposition of the inertial sensor is done without reference to any signalreceived from a source not mounted on or held by the user.
 59. Themethod of claim 58, wherein the drift of the relative position ororientation of the second inertial sensor relative to the first inertialsensor is corrected by measurements between devices on the user's headand devices elsewhere on the users body.